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Attorney Docket: 222 1/103 
Data Correlation and Analysis Tool 

Cross Reference to Related Applications 

This application is a continuation application from a U.S. patent application with 
5 serial number 09/413,692, filed October 7, 1999, which claims priority from U.S. patent 
number 5,995,941, filed September 15, 1997 and issued November 30, 1999, which claims 
priority from provisional U.S. patent application serial number 60/026,128, all three 
disclosures of which are incorporated herein, in their entirety, by reference. 

Background of the Invention 

10 The present invention related generally to apparatus and methods for correlating and 

analyzing data. More particularly, the invention relates to an apparatus and method for 
correlating data with respect to video images such that analysis on the data can be 
performed and video images associated with data segments can be readily retrieved. Many 
industries use optical systems in conjunction with various sensors to monitor 

15 environmental conditions. Examples of such industries are market research, security, and 
quality control. Various collection tools are employed in collecting the data and 
subsequently organizing the data. A common thread among these conmiercially available 
systems is that the data must be manually correlated with video data from the optical 
systems. For example, in the field of market research being performed on an oral 

20 presentation may include video images of the person making the oral presentation as well 
as feedback from the sample group. The feedback is generally obtained using electronic 
equipment which gauges the audience members reaction to the oral presentation. Once the 
presentation is complete the data from each of the audience members must then be 
correlated to that of the video segments. That is, when strong positive or negative reactions 

25 are attained, the relevant segment of the video must be located and data and the audience 
must be correlated therewith in order for the audience feedback to be meaningful. Often, this 
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is a tedious, time consuming manual task that delays feedback to the oral presenter from two 
(2) to six (6) weeks. 

The common practice is then to provide a written report detailing the analysis of the 
individual events that the audience found interesting in either a positive or negative manner. 
5 The report further identifies key responses that correlated to those events. Occasionally, 
along with the report a market research firm would provide the video tape of the oral 
presentation for further analysis by the presenter. 

These and other objects of the invention will be obvious and will appear hereinafter. 

Summary 

10 The aforementioned and other objects are achieved by the invention which provides a 

data correlation and analysis tool and a method associated therewith. The invention comprises a 
system for capturing a video image of a scene and for correlating data relative to the scene. 

In the data correlation and analysis tool, a collection means is used which receives the 
data from remote electronic sources and receives a video signal representative of the video image 

15 from a video camera 

The remote electronic sources that produce the data can be any of various types of 
sensors ranging from manually driven dial-type controls for human input to environmental 
controls, such as temperature gauges or flow controls in the case of process control, for example. 
Electrically connected to the collection means is a correlation means which correlates the 

20 data to the video signal. In this way the environmental responses are associated with the video 
images from the video camera substantially instantaneously. Additionally, statistical analysis can 
now be performed on the data to find particular segments of the video which were associated 
with highs or lows or any of various other points that may be of interest to us inquiring or 
analyzing the information. 

25 In further aspects, the invention provides methods in accord with the apparatus 

described above. The aforementioned and other aspects of the invention are evident in the 
drawings and in the description that follows. 
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Brief Description of the Drawings 

The foregoing and other objects of this invention, the various features thereof, as well 
as the invention itself, may be more fiiUy understood from the following description, when 
read together with the accompanying drawings in which: 
5 Figure 1 shows a block diagram of a data correlation and analysis tool in accordance 

with the invention; 

Figures 2A and 2B show block diagrams of additional applications of the data 
correlation and analysis tool shown in Figure 1; 

Figure 3 shows the data collection correlation tool and analysis tool of Figure 1 in 
10 greater detail; 

Figure 4 shows a table representative of correlated data as found in memory within 
the data correlation and analysis tool of Figure 3; 

Figure 5 shows a table representative of correlated data which would be found in a 
memory associated with the data correlation and analysis tool of Figure 1; and 
15 Figures 6 A and 6B shows sample user interfaces for a computer display presenting 

the correlated and analyzed data generated by the invention. 

Detailed Description 

While the present invention retains utiUty within a wide variety of data correlation and 
analysis systems, and data acquisition systems generally, and may be embodied in several different 
20 forms, it is advantageously employed in connection with market research systems. Though this is 
the form of the preferred embodiment and will be described as such, this embodiment should be 
considered illustrative and not restrictive. 

Figure 1 shows a block diagram of a data correlation and analysis system 10 being used for^ 
market research. The data correlation and analysis system 10 uses processing module 12 which 
25 has various inputs firom the environment. In this embodiment, a presenter 14 is making an oral 
presentation to a sample audience 24, 28. The presenter 14 could be interested in audience 
feedback for any of various reason: he/she has an upcoming important oral presentation and wants 
to know audience reaction; or has a new product or a series of products and is trying to ascertain 
public reaction to the product, for example. Additionally, the presenter 14 could be trying to gain 
30 judgement analysis on such topics as political, governmental, or legal presentations, for example. 
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In this embcxiiment, the presenter 14 speaks into a microphone 18 and has a video image 
c^tured by a video camera 20, both of which are fed into the processing module 12. One skilled in 
the art will realize that the presenter 14 need not be local for the invention to operate. The presenter 
14 can be local or offsite, and the presentation can be in real time or time delayed. Within the 
5 processing module 12, the audio signal from the microphone 18 and the video signal from the 
video camera 20 are transmitted into a correlation unit 22. 

Simultaneously therewith, an audience segment 24 inputs via response meters 26 their 
impression regarding the presentation being made by the presenter 14. One skilled in the art 
will realize that the audience can have varying population and demographics depending upon 

10 an amount of error is judged acceptable for the subsequent statistical analysis of their opinions. 

In practice, as the presenter 14 speaks the members of the first audience segment 24 
will judge positively or negatively in real time their impression of what is being said or done 
by the presenter 14. This impression is dialed into the response meters 26 by indicating on the 
response meter a numerical value representative of that impression. Each.member of the 

15 audience has a response meter 26 to ensure that individual opinions are monitored. One skilled 
in the art will realize that the response meter 26 can be an analog or digital device which is : . 
commonly available in the art. In this embodiment, an analog meter is used which has a dial 
with a predetermined range, such as from one to ten, where one equates to a poor opinion of 
the presentation and ten means the presentation is excellent. 

20 A member of the first audience segment 24 then uses the response meter 26 to dial in 

his/her impression of the presentation by selecting a value in that range. The response signal 
from the response meter 26 is fed back into the processing module 12 and into the correlation 
unit 22. where the response signal is correlated by time with the video signal and the audio 
signal. 

25 In a simple data correlation and analysis system, the aforedescribed system could be 

complete with each member of the audience dialing in his/her impressions via the response 
meter 26 and being correlated. The hereinafter described analysis can then be performed 
without need for any additional information. In this embodiment though, a second audience 
segment 28 is shown also having response meters 26. This second audience segment 28 can 

30 be used for any of numerous reasons. The market research firm may wish to know, for 
example, how do the audiences impressions change with respect to their angle to the 
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presenter, when they are in a separate room from the presenter 14, or in another city, inter 
alia. In this embodiment, the second audience segment 28 is located at an angle substantially 
different from that of the first audience segment 24. 

A second video camera 30 is then located near the second audience segnient 28. 
5 Although the second video camera 30 is optional, it does provide information on what is the 
actual image being perceived by the second audience segment 28. In this embodiment, the 
information obtained from these two groups can then be interpolated or extrapolated to many 
varying angles and distances from the presenter 14 with or without additional video feedback 
associated with each one. 

10 Another form of data from the environment of the audience segments 24, 28 is 

environmental sensory information itself. An example of such environmental sensory 
information is that of temperature shown by a thermometer 34. Thermometers, moisture 
meters, et cetera, can be disposed around each audience segment such that data of 
environmental conditions is then correlated via the correlation unit 22 to the video image 

15 providing additional feedback for the presenter 14. Such a layout could show that as an 
audience gets warmer their opinions of the presentation go down, for example. 

As previously stated, the various video, audio and data signals are input into the 
correlation unit 22. The correlated data 36 is then passed to an analysis unit 38 where the 
correlated data 36 is analyzed. Generally speaking, statistical analysis is used in the analysis. 

20 unit 38 to find points of interest in the presentation by the presenter 14. For example, the 

analysis unit 38 can determine all low points where the audience was in general agreement of 
having a negative impression. The presenter 14 would be interested in such information such 
that it can be removed from the presentation or altered to avoid such negative reactions. 

The form of the analysis can exceed simple averaging techniques. Numerous forms 

25 of statistical analysis can be done. For example, demographic information can be input into 
the analysis unit and can be stored in the correlation unit 22 or extemally in a computer 40. 
The correlated data 36 can then be analyzed by demographic information such that the 
reactions of males versus females, for example, at any point in time can be analyzed. 
Further, analysis can be done by age or any various other demographic criteria that had been 

30 requested of the audience segments 24, 28. 

In the case of using an external computer 40, analyzed correlated data can be passed 
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to the computer or the raw information can be passed to the computer 40. If much of the 
processing is to be performed by the computer 40, then one of the purposes of the analysis 
unit 38 would be digital signal processing such that signals are smoothed and noise is 
reduced within the correlated data. 
5 The computer 40 displays to user of the computer 40 through a graphical user 

interface ("GUF') 42 any statistical analysis that is requested as well as its associated video 
and audio segments. The GUI 42 can display the data in real time where the correlated data 
36 is essentially passed through the analysis unit 38 to the computer 40 and is displayed. Or, 
the GUI 42 can operate on archival information where the information is stored on a bulk 
10 storage device, such as a computer hard disk, and analysis is performed on the archived 
information. 

Figures 2A and 2B show additional embodiments of the invention. Figure 2A shows 
a data correlation and analysis system 50 being used for quality control where a product 54 
comes down a conveyor belt 52 to a quality control station having a camera 56 and a set of 

15 sensors 58. Depending on the type of product 54 that is being analyzed, the camera 56 can 
be a video camera which takes a video image of the product 54 in which case the video 
image is compared to a stored video image. Based upon statistical analysis, the data 
correlation and analysis system 50 then rejects or accepts the product 54. The camera 56 
could also be a sonar or x-ray device for examdning welds in manufactured devices. Further, 

20 in the case of welds, additional environmental information can include the temperature at 
which the weld was made for better quality control of the welds. 

Figure 2B shows another embodiment of the invention where the data correlation and 
analysis system 60 is being used for security purposes. In this embodiment, a video camera 
66 is located in a room 62 having a door 64 which is being monitored. An environmental 

25 sensor, in this embodiment a temperature gauge 68, is also located in the room. 

The video signal from the video camera 66 and temperature information from the 
thermometer 68 are passed back to the processing module 12 where each reading of the 
thermometer 68 and each frame or frames of the video are associated with time and analyzed. 
In this way, a statistical correlation can be found and statistical anomalies can be flagged to 

30 security personnel. For example, decreasing temperature can be correlated to the door 64 
being opened alerting security personnel to the door 64 having been opened and marking for 

6 



2221/103 



the security personnel the time of the decreased temperature and thus the entry. One skilled 
in the art will understand that numerous types such as door sensors and smoke sensors and 
quantities of sensors can be used to gain more information about the environment. 

Figure 3 shows an additional embodiment of the invention where a VCR 76 is included 
5 in the processing module 12. In this embodiment, the video signal from the video camera 20 is 
passed into the processing module 12 and into the correlation unit 22, but is simultaneously 
passed into video storage device, such as a VCR 76, a writable CD or other device without regard 
to whether the device is analog or digital. Likewise, the audio signal from the microphone 18 is 
passed into the correlation unit 22 and is also passed into the VCR 76. Since both the video 

10 signal and the audio signal in this embodiment are analog signals, the VCR 76 can create a video 
tape of the presentation in. a standard format, such as VHS for example. The presenter 14 can 
then have a full video tape of his/her presentation upon completion of the presentation. 

Substantially simultaneously, the second audience segment 28 is using the response 
meter 26 to dial in their feedback to the presentation as previously described. In this 

15 embodiment, the response meters 26 are again analog devices wherein an analog signal 

representative of the response is passed into the correlation unit 22. One skilled in.the art will 
realize that digital response meters can also be used. It should also be apparent that any of the 
response meters 26 can be wireless, and as such transport the response by RF to an antenna . 
connected to the processing module 12. Examples of such digital response meters are computer 

20 devices where a computer display is shown to the audience member and responses are either . 
input numerically via a keyboard, a touch-screen display is used, or any of various pointing 
devices. 

Within the correlation unit 22, the video signal, audio signal and the numerous 
response signals pass through analog to digital converters 70, 72, 78, 80 where the signals are 
25 converted into digital form. Digital signals can then be passed into the analysis unit 38 where 

4 

they are selectively stored on a memory device 74. A cache 21, as more fiiUy described 
hereinafter, can also be present in the correlation unit 22 and assist with passing the digital 
signals from the memory device 74. 

Depending on the implementation, the memory device 74 can be a magnetic disk or 
30 can be solid-state memory. 

The analysis unit 38 subsequently performs statistical analysis on the output of the 
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correlation unit 22 using a microprocessor 82. In this embodiment a summary video can be 
made where segments of the video information can be parsed based on interesting statistical 
correlations among the audience members thereby creating a sunmiary video having only 
segments which are of interests to the presenter 14. For example, for basic highlights or 
5 lowlights, the analysis can simply look for somewhat uniform highs and lows in the 
responses and transmit video segments common to those responses. 

If desired', demographics can be used to further refine the statistical analysis as 
previously described. The demographics can be input via the computer 40 such that ages, the 
sex and various other information relating to the audience members is known to the analysis 

10 unit 38 and video segments can be parsed based on demographic information. For example, 
the presenter 14 may be interested in knowing what the males thought of the presentation 
versus the females, or what different age groups felt about different segments of the 
presentation then the summary video would present the highlights and lowlights for the 
desired demographic group. 

15 In this embodiment, the mass storage device 74 is a high-capacity magnetic storage 

device. If the video is being archived on the mass storage device 74, the video must be 
compressed. The reason for this is that video images use a substantial amount of storage : 
capacity. Current video technology uses 30 frames per second where each frame has a 
normal resolution of 640 by 480 pixels. While this resolution can be altered, the. 

20 aforementioned resolution is considered to be an acceptable video image. 

The video is then generally in color and, therefore, has 24 bits of color associated 
with each pixel, eight bits per red, green, and blue ("RGB") color, creating approximately 1 
megabyte of video information per frame at 30 frames per second. One skilled in the art will 
realize that the resolutions and the storage capacities used in the preferred embodiment are 

25 dependent upon the current state of the art. Higher resolutions may be used as the art 
advances in the area of image compression, or lower cost, higher capacity bulk storage 
devices become available. 

For a one-hour video, this creates approximately 100 gigabytes of video information. 
This amount does not include audio or any of the response information necessary to do a 

30 proper analysis. Therefore, the video data must be compressed. In preferred embodiment, 
compression is performed within the correlation unit 22 such that the video is reduced to ten 
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frames per second thereby cutting the storage requirements by one third, though one skilled 
in the art will realize that that increasing the number of frames per second is advantageous. 

Figure 4 shows a table of the contents of a typical memory used in the data 
correlation and analysis system. In a preferred embodiment a multi-channel associative 
5 cache is used. The multi-channel associative cache is an associative memory model which 
associates multiple video, audio and response variables 94 and newly created variables 96. 
with the dimension of time. In Figure 4 there are multiple, up to 'n' video and audio stimulus 
streams that are gathered and stored within correlation unit 22. 

The multi-channel associative cache logs the location of the stored and synchronized 
10 digitized multiple video and audio frames, as well as associations in the real time process, 
i.e., the response variables and their responses with these frames, other processed variables 
and new variables. These associations are then ordered as a function of time. 

This associative memory then allows access to associative process variables 94, the 
new variables 96 as well as video and audio frames 92 by content as opposed to by physical 
15 address. Thus, analysis of the associated information is easily accessible. • . 

This is referred to as a multi-channel associative cache because there are multiple, 
separate streams of input. In the associative cache the process variable might be a set of 
"male" respondents who are responding to one or more of the video-audio stimulus. In this 
example there could be up to 100 male respondents during the video frames associated with 
20 the first time slice, ti. Any number of actual responders are possible. 

Figure 5 is another method by which the content of a typical memory can be 
displayed. Again, the multi-channel associative cache associates with time 100 multiple 
variables 102 (vl, v2, ... vm,) such as multiple video, audio or text, with process variables 
104 (pl,p2, . . . pO) such as responses, and environmental variables, and the newly created 
25 variables 106 (nvl, nv2, . . .nvy,) such as averages, for example. The newly created variables 
106 can be one variable, such as an average, or a multitude of variables, such as a set of 
values of how far each process variable is from the mean, for example. 

In this Figure, however, not only is a location designated in the table as a "floe" 
stored with each variable 102, the presence of separate objects (ol, o2...on) within each of 
30 the variables 102 is noted by the presence of a 1. In this way, the presenter 14 can also see 
how objects affected the response. For example, at a political rally the system or presenter can 
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designate as an object each nominee. The system would designate the nominees presence by a 1 
for the nominee's object in the video variable and the nominee's voice by a 1 for the nominee's 
object number in the audio variable. Thus, the campaign can determine, for example, how the 
politician was received when they were on stage versus when they were talking on stage. If a 
5 nominee's spouse was designated as an object, the campaign can have analyzed the effect of the 
spouse's presence on the nominee's acceptance, for example. 

The table can also convey the frequency of the variables 102 as is noted by the presence 
or absence of a "floe" in the table. Reading a column in the table allows the presenter to visually 
see the intermittent nature of the audio signal. In addition to communicating visually the 

10 frequency of events, the presence or the absence of a variable over time, can also assist the 

presenter to choose an appropriate time period it should instruct the analysis unit 38 to wait after 
the audio segment before analyzing the response. In the preferred embodiment, the analysis unit 
38 wiU allow for a lag time between the one or more audio-video variables 102 and the process 
variables 104 upon which it will analyze, but this can be overridden by the presenter. 

15 Further, although the table can store locations of the audio-video variables by a 

pointer, a representation of the video frame can also be stored in the table. The 
representation would normally be much smaller than the frame. 

Turning now to figures 6A and 6B which show control screens that allow the operator 
to control the automatic analysis and automatic creation of the summary audio- video which 

20 is shown above. In this screen, one or more processed variables are available to analyze 
along with an access to one or more audio-video variables. Analysis of a selected set of 
process variables are then available to be displayed along with its associated time series. . 
Video segments are also available to be searched to determine associated process variables 
and their resulting analysis. The video-audio search can be done via a video search frame or 

25 clip 1 18 or supplement of a video frame clip. The video-audio segments associated with the 
variables or analysis of the variables can be displayed in a video subwindow 110. One can even 
select a set of criteria, highs or lows for examples, so that analysis then done can 
automatically find the associated video segments and play them back or create a new video. 
A text search string 116 can also be used to locate segments of a video so that the 

30 assodated responses or their analyses can be retrieved, fiirth^ analyzed and used to create new 

stimulus for the research group. Text strings are usually a result of automatic analysis of the 
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audio streams and are also stored in the associative map. Alternative display forms are easily 
created that allow for the display of multiple process variables 138, 140, 142, 144, multiple 
analyzed processed variables as well as scrolling text windows 132 as the video is played 
130. 

5 Pop down windows allow for the control of what statistical analysis is to be 

performed on the selected process variables as well as preferences and other control 
parameters. These pop down windows also allow one to tell the system how to deal with 
sample data and summary video outputs and inputs. 

For example, the variable list 112 shows a list of the variables that might be used in a 

10 particular study. A user would select a process variable and the associative cache would 
return from the analysis unit 38 values for that variable. The analysis unit 38 would 
manipulate the data, such as smooth it and calculate highs and lows, inter alia, and present to 
the GUI 42 the audio data over time 120 and the response over time 122. Through the GUI 
42, the presenter 14 can tell the analysis unit 38 to create other new variables or tell the 

15 associative cache to find other associated variables. 

The invention may be embodied in other specific forms without departing from the 
spirit or essential characteristics thereof. The present embodiments are, therefore, to be . 
considered in all respects as illustrative and not restrictive, the scope of the invention being 
indicated by the appended claims rather than by the foregoing description, and all changes 

20 which come within the meaning and range of equivalency of the claims are therefore 
intended to be embraced therein. 



